Display device for processing user utterance and control method of display device

ABSTRACT

According to an embodiment disclosed in the specification, a display device may include a microphone, a display displaying a screen including a plurality of layers, a memory storing a plurality of application programs, and at least one processor displaying a first user interface (UI) for interacting with a user on a first layer among the plurality of layers, displaying a second UI for displaying information obtained by performing the interaction on a second layer among the plurality of layers, and displaying an image at least partly overlapping with the first UI and the second UI on a third layer among the plurality of layers.

TECHNICAL FIELD

Embodiments disclosed in the disclosure refer to a technology forprocessing a user utterance.

BACKGROUND ART

In addition to a conventional input method using a keyboard or a mouse,electronic devices have recently supported various input methods such asa voice input and the like. For example, the electronic devices such assmart phones or tablet PCs may receive a user voice and then may providea service that performs an operation corresponding to the received uservoice. The speech recognition service is being developed based on atechnology for processing a natural language.

Besides, the electronic devices may provide a feedback corresponding toa user utterance and then may receive a user input corresponding to thefeedback, thereby providing a service desired by a user.

DISCLOSURE Technical Problem

A display device may display both a UI for receiving a user utteranceand a screen for displaying a result of processing the user utterance ona display, when processing a function corresponding to the received userutterance. Accordingly, the display device may only process the userutterance associated with the first entered user utteranceintermittently; it may also be difficult to provide a feedback on thereceived user input. In other words, it may be difficult for the displaydevice to organically process a plurality of user inputs associated withone another.

According to various embodiments of the disclosure, the display devicemay organically process user utterances associated with one anotherthrough interaction with a user.

Technical Solution

According to an embodiment disclosed in the specification, a displaydevice may include a microphone, a display displaying a screen includinga plurality of layers, a memory storing a plurality of applicationprograms, and at least one processor displaying a first user interface(UI) for interacting with a user on a first layer among the plurality oflayers, displaying a second UI for displaying information obtained byperforming the interaction on a second layer among the plurality oflayers, and displaying an image at least partly overlapping with thefirst UI and the second UI on a third layer among the plurality oflayers. The at least one processor may be configured to display thefirst UI on the display, in a process of performing the interactionthrough the microphone and in a process of displaying a result ofperforming the interaction on at least one of the second UI and thethird layer.

Furthermore, according to an embodiment disclosed in the specification,a method may include displaying a first UI for interacting with a useron a first layer among a plurality of layers included in a screendisplayed on a display, receiving a first user input through amicrophone, and displaying a result corresponding to the first userinput, on a second UI for displaying information obtained by performingthe interaction on a second layer among the plurality of layers or on athird layer on which an image at least partly overlapping with the firstUI and the second UI is displayed.

Furthermore, according to an embodiment disclosed in the specification,a program for executing a method may be recorded in a recording medium.The method may include displaying a first UI for interacting with a useron a first layer among a plurality of layers included in a screendisplayed on a display, receiving a first user input through amicrophone, and displaying a result corresponding to the first userinput, on a second UI for displaying information obtained by performingthe interaction on a second layer among the plurality of layers or on athird layer on which an image at least partly overlapping with the firstUI and the second UI is displayed.

Advantageous Effects

According to embodiments disclosed in the specification, in the processof performing a specified function depending on a user utterance, adisplay device may display a UI for interacting with a user and a UI fordisplaying a result of performing the specified operation on a pluralityof layers. Accordingly, the display device may organically process aplurality of user utterances associated with one another whileminimizing the obscuration of the image of content, and may provide theuser with a user experience, such as talking to the display device.

Besides, a variety of effects directly or indirectly understood throughthe disclosure may be provided.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a user utterance processing systemaccording to various embodiments.

FIG. 2 is a block diagram illustrating a configuration of a displaydevice according to various embodiments of the disclosure.

FIG. 3 is a view illustrating a plurality of layers of a screendisplayed on a display according to various embodiments of thedisclosure.

FIG. 4 is a diagram illustrating a method in which a display devicedisplays a screen including a plurality of layers on a display,according to various embodiments.

FIG. 5 is a screen in which a display device changes a channel dependingon a user utterance, according to an embodiment.

FIG. 6 is a screen on which a display device provides video on demand(VOD) depending on a user utterance, according to an embodiment.

FIG. 7 is a screen on which a display device executes an app dependingon a user utterance, according to an embodiment.

FIG. 8 illustrates a screen, on which a volume according to a userutterance of a display device is changed, according to an embodiment.

FIG. 9 is a screen on which an option menu is displayed on a displaydepending on a user utterance of a display device, according to anembodiment.

FIG. 10 is a screen on which a display device makes a viewingreservation depending on a user utterance, according to an embodiment.

FIG. 11 is a screen on which a display device reserves an airline ticketdepending on a user utterance, according to an embodiment.

FIG. 12 is a screen on which a display device orders food depending on auser utterance, according to an embodiment.

FIG. 13 is a screen on which a display device displays weatherinformation depending on a user utterance, according to an embodiment.

FIG. 14 is a screen on which a display device provides specifiedinformation and then makes a viewing reservation depending on a userutterance, according to an embodiment.

With regard to description of drawings, the same or similar componentsmay be marked by the same or similar reference numerals.

MODE FOR INVENTION

Hereinafter, various embodiments of the disclosure will be describedwith reference to accompanying drawings. However, those of ordinaryskill in the art will recognize that modification, equivalent, and/oralternative on various embodiments described herein can be variouslymade without departing from the scope and spirit of the disclosure. Withregard to description of drawings, similar components may be marked bysimilar reference marks/numerals.

FIG. 1 is a diagram illustrating a user utterance processing systemaccording to various embodiments.

Referring to FIG. 1, a user utterance processing system 10 may include adisplay device 100 and a user utterance processing server 200. Thedisplay device 100 and the user utterance processing server 200 may beconnected over a communication network.

According to an embodiment, the display device 100 may provide contentto a user. For example, the display device 100 may provide the contentto the user by respectively outputting an image and a sound included inthe content through a display and a speaker. According to an embodiment,the user utterance processing system 10 may output the content stored ina memory or the content received from an external device.

According to an embodiment, the display device 100 may display a userinterface (UI) for providing information, on the display. For example,the display device 100 may display a UI for receiving a user utteranceon the display. For another example, the display device 100 may displaythe result of executing a specified operation on the UI. For example,the UI may be displayed on an image of the content displayed on thedisplay. In other words, at least part of the UI may be displayed whileoverlapping with the image of the content.

According to an embodiment, the display device 100 may provide aspecified service to the user. For example, the display device 100 mayprovide the specified service to a user through the installedapplication program (or app). In other words, the display device 100 mayexecute an app and then may provide a service corresponding to a userinput through the executed app.

According to an embodiment, the display device 100 may receive a userutterance for performing a specified function. According to anembodiment, the display device 100 may process the received userutterance to provide the specified service to the user. According to anembodiment, the display device 100 may process the received userutterance through the user utterance processing server 200. For example,the display device 100 may transmit the received user utterance to theuser utterance processing server 200 and then may receive information(e.g., a path rule) for performing the specified function from the userutterance processing server 200.

According to an embodiment, the user utterance processing device 200 mayprocess the user utterance. For example, the user utterance processingdevice 200 may receive a user utterance from the display device 100, mayprocess the received utterance, and may transmit the processed utteranceto the display device 100.

According to an embodiment, the user utterance processing device 200 mayprocess the received user utterance. According to an embodiment, theuser utterance processing device 200 may include an auto speechrecognition module (or ASR module) for converting a user utterance to atext and a natural language understanding module (or NLU module) fordetermining the intent of the user utterance.

According to an embodiment, the ASR module may generate a textcorresponding to the received user utterance. According to anembodiment, the NLU module may analyze the generated text to determine afunction corresponding to the user utterance. For example, the NLUmodule may analyze the text to grasp the user's intent and may determinethe function to be executed depending on the intent. In addition, theNLU module may extract a parameter needed to perform the determinedfunction from the text. Accordingly, the user utterance processingdevice 200 may generate (or select) information (e.g., a path rule)including the determined function and the extracted parameter.

The display device 100 may process the intermittently received userutterance. For example, the display device 100 may receive a user inputand may provide the result of performing a function corresponding to thereceived user input. In other words, the display device 100 may displayboth a UI for receiving a user utterance and the result of processingthe user utterance on the display. Accordingly, when the display device100 performs a function corresponding to the received user utterance, itmay be difficult to organically provide a feedback on the received userutterance. In addition, to execute an additional function associatedwith the first entered user utterance, the user may have theinconvenience of re-entering information included in the first entereduser utterance. The display device 100 according to various embodimentsof the disclosure may perform a specified function in a dialog formthrough the organic interaction with a user.

FIG. 2 is a block diagram illustrating a configuration of a displaydevice according to various embodiments of the disclosure.

Referring to FIG. 2, the display device 100 may include a communicationinterface 110, a microphone 120, a display 130, a speaker 140, a memory150, and a processor 160.

According to an embodiment, the communication interface 110 may beconnected to an external device to communicate with the external device.For example, the communication interface 110 may be connected to theuser utterance processing server 200 to transmit a user utterance to theuser utterance processing server 200. According to an embodiment, thecommunication interface 110 may include a wired communication interfaceand a wireless interface. For example, the wired communication interface(e.g., High-Definition Multimedia Interface (HDMI), digital visualinterface (DVI) interface, video graphics array (VGA) interface, or thelike) may be connected to the external device such as a set-top box, adigital versatile disc (DVD) player, and the like via a wire to receivecontent. The wireless communication interface may be connected to theuser utterance processing server 200 through an access point (AP) toprocess the user utterance.

According to an embodiment, the microphone 120 may receive the userutterance from a user, For example, the microphone 120 may receive theuser utterance through a speech input system. In this specification, theuser utterance may be referred to as a voice signal.

According to an embodiment, the display 130 may display an image (or avideo image). For example, the display 130 may display the image ofcontent. For another example, the display 130 may display the userinterface (UI) of the executed app. For example, the UI may be a graphicuser interface (GUI).

According to an embodiment, the speaker 140 may output sound. Forexample, the speaker 140 may output the sound of content. For anotherexample, the speaker 140 may output voice sound. For example, the voicesound may be a feedback voice corresponding to a user input.

According to an embodiment, the memory 150 may include a plurality ofapps 151, 153, and 155. For example, the plurality of apps 151, 153, and155 may be programs for performing a function corresponding to the userinput. (including volatile and non-volatile memory)

According to an embodiment, the first app 151 may be an app forproviding the specified content. For example, the first app 151 mayprovide video on demand (VOD) corresponding to a user input. Accordingto an embodiment, the second app 153 may be an app for obtaining thespecified information. For example, the second app 153 may obtaininformation about the result of a sport game, local weather information,and the like. According to an embodiment, the third app 155 may be anapp for purchasing a product, or the like. For example, a user may booktravel products or may order food, through the third app 155.

According to an embodiment, the processor 160 may be operativelyconnected to the communication interface 110, the microphone 120, thedisplay 130, the speaker 140, and the memory 150 to control the overalloperation of the display device 100.

According to an embodiment, the processor 160 may output content. Forexample, the processor 160 may output images and sound, which areincluded in the content received from the external device through thecommunication interface 110, through the display 130 and the speaker140, respectively. According to an embodiment, the processor 160 mayoutput content corresponding to a user input.

According to an embodiment, the processor 160 may display a UI forproviding information on the display 130. For example, the processor 160may display a UI for interacting with a user. In addition, the processor160 may provide a feedback or a result (or information) corresponding toa user utterance.

According to an embodiment, the processor 160 may display a screenincluding a plurality of layers on the display 130. For example, theprocessor 160 may display a UI for interacting with a user, a UI fordisplaying information obtained by performing the interaction, a UI fordisplaying the feedback on a user input, and the image of content, onthe plurality of layers, respectively.

According to an embodiment, the processor 160 may compose the UI forinteracting with a user, the UI for displaying information obtained byperforming the interaction, the UI for displaying the feedback on a userinput, and the image of content, which are respectively displayed on theplurality of layers, and then may display the composed result on thedisplay 130. Accordingly, for example, the UI for interacting with auser, the UI for displaying information obtained by performing theinteraction, and the UI for displaying the feedback on a user input,which are respectively displayed on the plurality of layers, may bedisplayed in a specified region of a screen on which the image ofcontent is displayed.

According to an embodiment, the processor 160 may include a userutterance processing module 161 and an execution manager module 163.

According to an embodiment, the user utterance processing module 161 mayprocess the user utterance (or the first user utterance) receivedthrough the microphone 120. For example, the user utterance processingmodule 161 may transmit the received user utterance to the userutterance processing server 200 and may receive information forperforming a function corresponding to the user utterance from the userutterance processing server 200. In other words, the user utteranceprocessing module 161 may process the user utterance through the userutterance processing server 200.

According to an embodiment, the user utterance processing module 161 mayprovide the feedback on the received user utterance. For example, whenthe user utterance includes information insufficient to execute thespecified function, the user utterance processing module 161 may providethe feedback on the user utterance. For example, the feedback mayinclude an input request for the insufficient information. According toan embodiment, the user utterance processing module 161 may receive theuser utterance (or the second user utterance) corresponding to thefeedback. The user utterance processing module 161 may process thereceived user utterance.

According to an embodiment, the processor 160 may display a UI (or thefirst UI) for interacting with a user in one layer (or the first layer)among a plurality of layers included in the screen displayed on thedisplay 130. For example, the interaction may include a user utteranceand a feedback. According to an embodiment, the processor 160 maydisplay an indicator for indicating the state of the user utteranceprocessing module 161 on the UI. For example, when the user utteranceprocessing module 161 is receiving a user utterance, the processor 160may display the indicator in the first state (e.g., blue). When the userutterance processing module 161 provides a feedback, the processor 160may display the indicator in the second state (e.g., white).Accordingly, the user may recognize a process of the interaction throughthe indicator.

According to an embodiment, the processor 160 may display a UI (or thesecond UI) for outputting the feedback in one layer (or the secondlayer) among the plurality of layers included in the screen displayed onthe display 130. According to an embodiment, the processor 160 mayoutput the feedback through the UI speaker 130. In other words, theprocessor 160 may output the same content as the feedback output throughthe UI.

According to an embodiment, the processor 160 may execute a functioncorresponding to the user utterance processed through the user utteranceprocessing module 161. For example, the processor 160 may display theimage of content corresponding to the processed user utterance in onelayer (or the third layer) of the plurality of layers included in thescreen displayed on the display 130. For example, the user utterance maybe an input for changing a channel. For another example, the userutterance may be an input for outputting the specified content. Foranother example, the processor 160 may transmit information about theuser utterance processed through the user utterance processing module161 to the execution manager module 163.

According to an embodiment, the execution manager module 163 may performthe specified function, using the information received from the userutterance processing module 161. According to an embodiment, theexecution manager module 163 may select at least one of a plurality ofapps stored in the memory 150 based on the information and may executethe selected app to perform the specified function. For example, thespecified function may be a function corresponding to a user input.

According to an embodiment, the processor 160 may display a UI (or thethird UI) for displaying the result of the function performed by theexecution manager module 163, in one layer (or the fourth layer) of theplurality of layers included in the screen displayed on the display 130.For example, the processor 160 may obtain information by performing afunction corresponding to the user utterance and then may display theobtained information on a UI for displaying the result of the functionperformed by the execution manager module 163. According to anembodiment, the processor 160 may display the UI (or the fourth UI) ofan application executed to perform a function corresponding to a userinput in one layer (or the third layer) of the plurality of layersincluded in the screen displayed on the display 130. For example, theprocessor 160 may display the UI of an application executed to perform afunction corresponding to a user input in the layer where the image ofthe content is displayed, instead of the image of the content.

According to an embodiment, the processor 160 may receive the first userutterance through the microphone 120. The processor 160 may process thereceived first user utterance. For example, the processor 160 maytransmit a user utterance to the user utterance processing server 200through the communication interface 110 and may receive information forexecuting a function corresponding to the user utterance. According toan embodiment, the processor 160 may display the result corresponding tothe first user utterance on the display 130, using the receivedinformation. For example, the processor 160 may display the result inone layer (e.g., the second UI, which is displayed in the second layer,or the third layer) of the plurality of layers.

According to an embodiment, in the process of performing the interactionthrough the microphone 120 and in the process of displaying the resultof the interaction on one layer of a plurality of layers, the processor160 may display a UI for performing the interaction on the display 130.In other words, in the process of processing a user utterance, theprocessor 160 may display a UI for performing interaction with a user.Accordingly, the user may continuously enter a user input necessary forthe display device 100 to operate.

According to an embodiment, the display device 100 may provide the userwith a user experience (UX) in a conversational form by receiving thefeedback on the received user utterance and the user input correspondingto the feedback.

FIG. 3 is a view illustrating a plurality of layers of a screendisplayed on a display according to various embodiments of thedisclosure.

Referring to FIG. 3, the display device 100 may display a screenincluding a plurality of layers on a display. For example, the pluralityof layers may include a first layer 310, a second layer 320, a thirdlayer 330, and a fourth layer 340.

According to an embodiment, the display device 100 may display a firstUI 310 a (e.g., the first UI of FIG. 2) for interacting with a user onthe first layer 310 among a plurality of layers. For example, the firstUI 310 a may be displayed in a partial region of the first layer 310.According to an embodiment, the display device 100 may display anindicator indicating the state of the display device 100 on the first UI310 a. Besides, the display device 100 may display the received userutterance and a reply to the user utterance, in the first UI 310 a.

According to an embodiment, the display device 100 may display a secondUI 320 a (e.g., the second UI of FIG. 2) for displaying the feedback ona user input, on the second layer 320 (e.g., the second layer of FIG. 2)among the plurality of layers. For example, the second UI 320 a may bedisplayed in a partial region of the second layer 320. For example, thefeedback may include at least one object capable of being selecteddepending on a user input. According to an embodiment, the displaydevice 100 may receive a user input for a feedback through a microphone.For example, the user input may be an input to select one of at leastone object capable of being selected.

According to an embodiment, the display device 100 may display a thirdUI 330 a (e.g., the third UI of FIG. 2) for displaying informationobtained by performing interaction with a user, on the third layer 330(e.g., the fourth layer of FIG. 2) among the plurality of layers. Forexample, the third UI 330 a may be displayed in a partial region of thethird layer 330. According to an embodiment, the display device 100 mayobtain information through an application for performing a functioncorresponding to a user input and may display the obtained informationon the third UI 330 a. According to an embodiment, the display device100 may display an image 340 a on the fourth layer 340 (e.g., the thirdlayer of FIG. 2) among the plurality of layers. For example, the image340 a may include an image of content or the fourth UI (e.g., the fourthUI of FIG. 3) of the executed application.

According to an embodiment, the display device 100 may display thesequentially stacked first layer 310 to fourth layer 340 on a display.In other words, the display device 100 may display the first to fourthlayers 310 to 340 on the display based on the stacked order. Forexample, the first UI 310 a, the second UI 320 a, and the third UI 330 arespectively displayed on the first to third layers 310 to 330 may bedisplayed to be superimposed with the content image 340 a in a specifiedregion of the screen on which the content image 340 a displayed on thefourth layer 340 is displayed. The first UI 310 a, the second UI 320 a,and the third UI 330 a displayed to be superimposed on the screen may bedisplayed with a specified transparency on a screen where the contentimage 340 a is displayed as a whole. According to an embodiment, thedisplay device 100 may selectively display the second layer 320 and thefourth layer 330 on the display.

FIG. 4 is a diagram illustrating a method in which a display devicedisplays a screen including a plurality of layers on a display,according to various embodiments.

Referring to FIG. 4, the display device 100 may interact with a user andmay display the result of the interaction on a display.

According to an embodiment, in screen 410, the display device 100 maydisplay an image 401 on the display. For example, the image 401 mayinclude the image of content or the UI of the executed app.

According to an embodiment, in screen 420, the display device 100 maydisplay the image 401 and a first UI 403 for interacting with the user,on the display. For example, when the display device 100 receives a userinput in screen 410, the display device 100 may display the first UI 403on the display. For example, the first user input may be an input forperforming a specified function. The first user input may be receivedvia a microphone.

According to an embodiment, in screen 430, the display device 100 maydisplay the image 401, the first UI 403, and a second UI 405 fordisplaying a feedback on the first user input, on the display. Forexample, when the display device 100 receives the first user inputincluding information insufficient to perform the specified function inscreen 420, the display device 100 may display the feedback on the firstuser input, on the second UI 405. For example, the feedback may be toobtain the result corresponding to the first user input. According to anembodiment, the display device 100 may receive a second user inputassociated with the feedback.

According to an embodiment, when a user input (e.g., the second userinput) is insufficient to perform the specified function, the displaydevice 100 may provide the user with a plurality of feedbacks on theuser input received through the display. In other words, the displaydevice 100 may provide the user with the plurality of feedbacks toperform the specified function (1). According to an embodiment, when itis difficult to perform the specified function through the feedbacks,the display device 100 may display the image on the display to be thesame as screen 410 (2).

According to an embodiment, in screen 440, the display device 100 maydisplay the image 401, the first UI 403, and a third UI 407 fordisplaying the result of performing the function corresponding to a userinput, on the display. For example, in screen 420, the display device100 may display the result of performing the function corresponding tothe first user input, on the third UI 407 (3). For another example, inscreen 430, the display device 100 may display the result of performingthe function corresponding to the second user input, on the third UI 407(4). According to an embodiment, when the display device 100 completesthe provision of the result, the display device 100 may display an imageon the display to be the same as screen 410 (5). According to anembodiment, the display device 100 may display information obtainedthrough app for performing a function corresponding to a user input, onthe third UI 407.

According to an embodiment, in screen 450, the display device 100 maydisplay an image 409 corresponding to a user input on the display. Forexample, in screen 420, the display device 100 may display the image 409corresponding to the first user input on the display (6). For anotherexample, in screen 430, the display device 100 may display the image 409corresponding to the second user input on the display (7). According toan embodiment, the display device 100 may display an image of contentcorresponding to a user input or the UI of an app capable of performinga function corresponding to the user input on the display.

FIG. 5 is a screen in which a display device changes a channel dependingon a user utterance, according to an embodiment.

Referring to FIG. 5, the display device 100 may change a channeldepending on a user utterance to change the image of content displayedon a display (e.g., the display 130 in FIG. 2).

According to an embodiment, in screen 510, the display device 100 mayreceive a first user input for changing a channel called “sportschannel”. For example, the first user input may include informationinsufficient to specify a channel. According to an embodiment, thedisplay device 100 may display a first UI 503 (e.g., the first UI 310 ain FIG. 3) for interacting with a user, on a screen on which an image501 of content (e.g., the image 340 a in FIG. 3) is displayed. Forexample, the image 501 of content may be displayed on the fourth layer(e.g., the fourth layer 340 in FIG. 3). The first UI 503 may bedisplayed on the first layer (e.g., the first layer 310 in FIG. 3).According to an embodiment, the display device 100 may display anindicator 503 a indicating the state of receiving the first user inputand a text 503 b corresponding to the first user input (e.g., a userutterance), on the first UI 503. For example, the indicator 503 a may bedisplayed in a first state (e.g., blue).

According to an embodiment, in screen 520, the display device 100 maydisplay a second UI 505 (e.g., the second UI 320 a in FIG. 3) fordisplaying a feedback on the first user input, on a screen on which theimage 501 of content is displayed. According to an embodiment, thedisplay device 100 may display a feedback 505 a including a channelcapable of being selected based on the first user input, on the secondUI 505. The second UI 505 may be displayed on the second layer (e.g.,the second layer 320 of FIG. 3). According to an embodiment, the displaydevice 100 may display a text 503 c for the reply of “What channel isit?” to the first user input on the first UI 503 and then may output thereply through a speaker. According to an embodiment, the display device100 may display the indicator 503 a for indicating a state of outputtingthe reply, on the first UI 503. For example, the indicator 503 a may bedisplayed in a second state (e.g., white).

According to an embodiment, in screen 530, the display device 100 mayreceive a second user input called “NBC Sports” associated with thefeedback 505 a. According to an embodiment, the display device 100 maydisplay the indicator 503 a indicating the state of receiving the seconduser input and a text 503 d corresponding to the second user input(e.g., a user utterance), on the first UI 503.

According to an embodiment, in screen 540, the display device 100 mayoutput an image 507 of content of NBC sports channel corresponding tothe second user input.

FIG. 6 is a screen on which a display device provides video on demand(VOD) depending on a user utterance, according to an embodiment.

Referring to FIG. 6, the display device 100 may output VOD depending ona user utterance to change the image of content displayed on a display(e.g., the display 130 in FIG. 2).

According to an embodiment, in screen 610, the display device 100 mayreceive the first user input for playing the VOD of “Show Star Wars inCOOK!”. For example, the first user input may include informationinsufficient to specify an episode to be played. According to anembodiment, the display device 100 may display a first UI 603 (e.g., thefirst UI 310 a in FIG. 3) for interacting with a user, on a screen onwhich an image 601 of content (e.g., the image 340 a in FIG. 3) isdisplayed. According to an embodiment, the display device 100 maydisplay an indicator 603 a indicating the state of receiving the firstuser input and a text 603 b corresponding to the first user input (e.g.,a user utterance), on the first UI 603.

According to an embodiment, in screen 620, the display device 100 maydisplay a second UI 605 (e.g., the second UI 320 a in FIG. 3) fordisplaying a feedback on the first user input, on a screen on which theimage 601 of content is displayed. According to an embodiment, thedisplay device 100 may display a feedback 605 a including an episodecapable of being selected based on the first user input, on the secondUI 605. According to an embodiment, the display device 100 may display atext 603 c corresponding to the reply of “What episode is it?” to thefirst user input, on the first UI 603 and then may output the replythrough a speaker. According to an embodiment, the display device 100may display the indicator 603 a for indicating a state of outputting thereply, on the first 603.

According to an embodiment, in screen 630, the display device 100 mayreceive a second user input called “Episode 2” associated with thefeedback 605 a. According to an embodiment, the display device 100 maydisplay the indicator 603 a indicating the state of receiving the seconduser input and a text 603 d corresponding to the second user input(e.g., a user utterance), on the first UI 603.

According to an embodiment, in screen 640, the display device 100 mayoutput an image 607 of content of “Star Wars Episode 2” corresponding tothe second user input.

FIG. 7 is a screen on which a display device executes an app dependingon a user utterance, according to an embodiment.

Referring to FIG. 7, the display device 100 may execute an app dependingon a user utterance to change the image of content displayed on adisplay (e.g., the display 130 of FIG. 2).

According to an embodiment, in screen 710, the display device 100 mayreceive the first user input for executing the app of “Execute CNNapp!”. For example, the first user input may include informationinsufficient to specify an app to be executed. According to anembodiment, the display device 100 may display a first UI 703 (e.g., thefirst UI 310 a in FIG. 3) for interacting with a user, on a screen onwhich an image 701 of content (e.g., the image 340 a in FIG. 3) isdisplayed. According to an embodiment, the display device 100 maydisplay an indicator 703 a indicating the state of receiving the firstuser input and a text 703 b corresponding to the first user input (e.g.,a user utterance), on the first UI 703.

According to an embodiment, in screen 720, the display device 100 maydisplay a second UI 705 (e.g., the second UI 320 a in FIG. 3) fordisplaying a feedback on the first user input, on a screen on which theimage 701 of content is displayed. According to an embodiment, thedisplay device 100 may display a feedback 705 a including an app capableof being selected based on the first user input, on the second UI 705.According to an embodiment, the display device 100 may display a text703 c corresponding to the reply of “What app is it?” to the first userinput, on the first UI 703 and then may output the reply through aspeaker. According to an embodiment, the display device 100 may displaythe indicator 703 a for indicating a state of outputting the reply, onthe first UI 703.

According to an embodiment, in screen 730, the display device 100 mayreceive a second user input called “CNN News” associated with thefeedback. According to an embodiment, the display device 100 may displaythe indicator 703 a indicating the state of receiving the second userinput and a text 703 d corresponding to the second user input (e.g., auser utterance), on the first UI 703.

According to an embodiment, in screen 740, the display device 100 mayoutput an image 707 of content of “CNN news” corresponding to the seconduser input. Accordingly, the display device 100 may execute an app(e.g., CNN news app) desired by a user.

FIG. 8 illustrates a screen, on which a volume according to a userutterance of a display device is changed, according to an embodiment.

Referring to FIG. 8, the display device 100 may prevent a volume frombeing changed beyond a specified size depending on a user utterance forchanging the volume.

According to an embodiment, in screen 810, the display device 100 mayreceive the first user input for changing the volume of “Volume 10”. Thefirst user input may include, for example, information enough to performan operation of changing the volume but may include a request forchanging the volume of the display device 100 to a specified magnitude(e.g., 50) or more. According to an embodiment, the display device 100may display a first UI 803 (e.g., the first UI 310 a in FIG. 3) forinteracting with a user, on a screen on which an image 801 of content(e.g., the image 340 a in FIG. 3) is displayed. According to anembodiment, the display device 100 may display an indicator 803 aindicating the state of receiving the first user input and a text 803 bcorresponding to the first user input (e.g., a user utterance), on thefirst UI 803.

According to an embodiment, in screen 820, the display device 100 maydisplay a second UI 805 (e.g., the second UI 320 a in FIG. 3) fordisplaying a feedback on the first user input, on a screen on which theimage 801 of content is displayed. According to an embodiment, thedisplay device 100 may display a feedback 805 a for identifying a changein volume based on the first user input, on the second UI 805. Accordingto an embodiment, the display device 100 may display a text 803 ccorresponding to the reply of “It will be noisy. Is it really?” to thefirst user input, on the first UI 803 and then may output the replythrough a speaker. According to an embodiment, the display device 100may display the indicator 803 a for indicating a state of outputting thereply, on the first UI 803.

According to an embodiment, in screen 830, the display device 100 mayreceive a second user input called “No” associated with the feedback 805a. According to an embodiment, the display device 100 may display theindicator 803 a indicating the state of receiving the second user inputand the text 803 d corresponding to the second user utterance (e.g., auser utterance).

According to an embodiment, in screen 840, the display device 100 maynot change the sound depending on the second user input.

FIG. 9 is a screen on which an option menu is displayed on a displaydepending on a user utterance of a display device, according to anembodiment.

Referring to FIG. 9, the display device 100 may change the specifiedoption by displaying an option menu on the display depending on a userutterance.

According to an embodiment, in screen 910, the display device 100 mayreceive a first user input for displaying the option menu called “expertsettings”. For example, the first user input may include informationinsufficient to specify the option menu. According to an embodiment, thedisplay device 100 may display a first UI 903 (e.g., the first UI 310 ain FIG. 3) for interacting with a user, on a screen on which an image901 of content (e.g., the image 340 a in FIG. 3) is displayed. Accordingto an embodiment, the display device 100 may display an indicator 903 aindicating the state of receiving the first user input and a text 903 bcorresponding to the first user input (e.g., a user utterance), on thefirst UI 903.

According to an embodiment, in screen 920, the display device 100 maydisplay a second UI 905 (e.g., the second UI 320 a in FIG. 3) fordisplaying a feedback on the first user input, on a screen on which theimage 901 of content is displayed. According to an embodiment, thedisplay device 100 may display a feedback 905 a including an optioncapable of being selected based on the first user input, on the secondUI 905. According to an embodiment, the display device 100 may display atext 903 c corresponding to the reply of “What setting is it?” to thefirst user input, on the first UI 903 and then may output the replythrough a speaker. According to an embodiment, the display device 100may display the indicator 903 a for indicating a state of outputting thereply, on the first UI 903.

According to an embodiment, in screen 930, the display device 100 mayreceive a second user input called “screen settings” associated with thefeedback 905 c. According to an embodiment, the display device 100 maydisplay the indicator 903 a indicating the state of receiving the seconduser input and a text corresponding to the second user utterance (e.g.,a user utterance).

According to an embodiment, in screen 940, the display device 100 maydisplay a screen setting menu 907 corresponding to the second user inputon a screen on which an image 901 of content is displayed.

FIG. 10 is a screen on which a display device makes a viewingreservation depending on a user utterance, according to an embodiment.

Referring to FIG. 10, the display device 100 may perform an operationfor a viewing reservation depending on a user utterance.

According to an embodiment, in screen 1010, the display device 100 mayreceive a first user input for making a viewing reservation saying that“Please book infinite challenges”. For example, the first user input mayinclude information insufficient to reserve for viewing. According to anembodiment, the display device 100 may display a first UI 1003 (e.g.,the first UI 310 a in FIG. 3) for interacting with a user, on a screenon which an image 1001 of content (e.g., the image 340 a in FIG. 3) isdisplayed. According to an embodiment, the display device 100 maydisplay an indicator 1003 a indicating the state of receiving the firstuser input and a text 1003 b corresponding to the first user input(e.g., a user utterance), on the first UI 1003.

According to an embodiment, in screen 1020, the display device 100 maydisplay a second UI 1005 (e.g., the second UI 320 a in FIG. 3) fordisplaying a feedback on the first user input, on a screen on which theimage 1001 of content is displayed. According to an embodiment, thedisplay device 100 may output a feedback 1005 a including channelinformation capable of being reserved for viewing based on the firstuser input, on the second UI 1005. According to an embodiment, thedisplay device 100 may display a text 1003 c corresponding to the replyof “What channel is it?” to the first user input, on the first UI 1003and then may output the reply through a speaker. According to anembodiment, the display device 100 may display the indicator 1003 a forindicating a state of outputting the reply, on the first UI 1003.

According to an embodiment, in screen 1030, the display device 100 mayreceive a second user input called “NBC next Saturday” associated withthe feedback 1005 a. According to an embodiment, the display device 100may display the indicator 1003 a indicating the state of receiving thesecond user input and a text 1003 d corresponding to the second userinput (e.g., a user utterance), on the first UI 1003.

According to an embodiment, in screen 1040, when completing the viewingreservation corresponding to the second user input, the display device100 may display a message saying that “viewing is reserved” on thesecond UI 1005 and may output the message through a speaker.

FIG. 11 is a screen on which a display device reserves an airline ticketdepending on a user utterance, according to an embodiment.

Referring to FIG. 11, the display device 100 may perform an operation ofreserving an airline ticket depending on a user utterance.

According to an embodiment, in screen 1110, the display device 100 mayreceive a first user input for reserving an airline ticket, which iscalled the “reservation for Jeju-do airline ticket”. For example, thefirst user input may include information insufficient to reserve.According to an embodiment, the display device 100 may display a firstUI 1103 (e.g., the first UI 310 a in FIG. 3) for interacting with auser, on a screen on which an image 1101 of content (e.g., the image 340a in FIG. 3) is displayed. According to an embodiment, the displaydevice 100 may display an indicator 1103 a indicating the state ofreceiving the first user input and a text 1103 b corresponding to thefirst user input (e.g., a user utterance), on the first UI 1103.

According to an embodiment, in screen 1120, the display device 100 maydisplay a second UI 1105 (e.g., the second UI 320 a in FIG. 3) fordisplaying a feedback on the first user input, on a screen on which theimage 1101 of content is displayed. According to an embodiment, thedisplay device 100 may output a first feedback 1105 a for making arequest for an itinerary input based on the first user input on thesecond UI 1105. According to an embodiment, the display device 100 maydisplay a text 1103 c corresponding to “What is your itinerary?” for thefirst user input, on the first UI 1103 and then may output the replythrough a speaker. According to an embodiment, the display device 100may display the indicator 1103 a for indicating a state of outputtingthe reply, on the first UI 1103. According to an embodiment, the displaydevice 100 may receive a second user input including information aboutitineraries.

According to an embodiment, in screen 1130, the display device 100 mayoutput a second feedback 1105 b including an airline ticket capable ofbeing selected based on the second user input on the second UI 1105.According to an embodiment, the display device 100 may display the text1103 d corresponding to the reply of “What flight do you use?” to thesecond user input, on the first UI 1103 and then may output the replythrough a speaker. According to an embodiment, the display device 100may receive a third user input to select an airline ticket.

According to an embodiment, in screen 1140, the display device 100 mayoutput a third feedback 1105 e including a request for identifying apayment based on the third user input. According to an embodiment, thedisplay device 100 may display the text 1103 e corresponding to thereply of “Do you want to make a payment?” to the third user input, onthe first UI 1103 and then may output the reply through a speaker.According to an embodiment, the display device 100 may receive a fourthuser input to identifying a payment.

According to an embodiment, in screen 1150, when completing the airlineticket reservation corresponding to the fourth user input, the displaydevice 100 may display a text 1103 f corresponding to a message sayingthat “airline ticket is reserved” on the second UI 1105 and may outputthe message through a speaker.

FIG. 12 is a screen on which a display device orders food depending on auser utterance, according to an embodiment.

Referring to FIG. 12, the display device 100 may perform an operation ofordering food depending on a user utterance.

According to an embodiment, in screen 1210, the display device 100 mayreceive the first user input for ordering the food saying that “Pleaseorder pizza!”. For example, the first user input may include informationinsufficient to order food. According to an embodiment, the displaydevice 100 may display a first UI 1203 (e.g., the first UI 310 a in FIG.3) for interacting with a user, on a screen on which an image 1201 ofcontent (e.g., the image 340 a in FIG. 3) is displayed. According to anembodiment, the display device 100 may display an indicator 1203 aindicating the state of receiving the first user input and a text 1203 bcorresponding to the first user input (e.g., a user utterance), on thefirst UI 1203.

According to an embodiment, in screen 1220, the display device 100 maydisplay a second UI 1205 (e.g., the second UI 320 a in FIG. 3) fordisplaying a feedback on the first user input, on a screen on which theimage 1201 of content is displayed. According to an embodiment, thedisplay device 100 may output a first feedback 1205 a including a foodmenu on the second UI 1105 based on the first user input. According toan embodiment, the display device 100 may display a text 1203 ccorresponding to the reply of “What menu would you like to eat?” to thefirst user input, on the first UI 1103 and then may output the replythrough a speaker. According to an embodiment, the display device 100may display the indicator 1203 a for indicating a state of outputtingthe reply, on the first UI 1203. According to an embodiment, the displaydevice 100 may receive a second user input to select a menu.

According to an embodiment, in screen 1230, the display device 100 mayoutput a second feedback 1205 b for identifying a shipping address basedon the second user input on the second UI 1105. According to anembodiment, the display device 100 may display the text 1203 dcorresponding to the reply of “Is the shipping address correct?” to thesecond user input, on the first UI 1203 and then may output the replythrough a speaker. According to an embodiment, the display device 100may receive a third user input to identify the shipping address.

According to an embodiment, in screen 1240, the display device 100 mayoutput a third feedback 1205 c including a request for identifying apayment based on the third user input. According to an embodiment, thedisplay device 100 may display the text 1203 e corresponding to thereply of “Do you want to make a payment?” to the third user input, onthe first UI 1203 and then may output the reply through a speaker.According to an embodiment, the display device 100 may receive a fourthuser input to identifying a payment.

According to an embodiment, in screen 1250, when completing the foodorder corresponding to the fourth user input, the display device 100 maydisplay a message saying that “Delivery within 30 minutes” on the secondUI 1205 and may output the message through a speaker. Accordingly, thedisplay device 100 may perform an operation of ordering the food desiredby the user.

FIG. 13 is a screen on which a display device displays weatherinformation depending on a user utterance, according to an embodiment.

Referring to FIG. 13, the display device 100 may provide a user withinformation associated with the content image displayed on a display,depending on a user utterance.

According to an embodiment, in screen 1310, the display device 100 mayreceive a first user input for providing weather information called“Busan weekend weather”. For example, the first user input may include arequest for information associated with an image 1301 of contentdisplayed on the display. According to an embodiment, the display device100 may display a first UI 1303 (e.g., the first UI 310 a in FIG. 3) forinteracting with a user, on a screen on which the image 1301 of content(e.g., the image 340 a in FIG. 3) is displayed. According to anembodiment, the display device 100 may display an indicator 1303 aindicating the state of receiving the first user input and a text 1303 bcorresponding to the first user input (e.g., a user utterance), on thefirst UI 1303.

According to an embodiment, in screen 1320, the display device 100 maydisplay a third UI 1305 (e.g., the third UI 330 a in FIG. 3) fordisplaying the result of the first user input, on a screen on which theimage 1301 of content is displayed. For example, the second UI 1305 maybe displayed on the third layer (e.g., the third layer 330 of FIG. 3).According to an embodiment, the display device 100 may displayinformation 1305 a about Busan weather based on the first user input onthe third UI 1305. According to an embodiment, the display device 100may display a text 1303 c for the reply of “it is sunny in Busan thisweek!” to the first user input, on the first UI 1303 and then may outputthe reply through a speaker. According to an embodiment, the displaydevice 100 may display the indicator 1303 a for indicating a state ofoutputting the reply, on the first 1303.

According to an embodiment, in screen 1330, when completing theprovision of information corresponding to the first user input, thedisplay device 100 may display only the image 1301 of content on thedisplay.

FIG. 14 is a screen on which a display device provides specifiedinformation and then makes a viewing reservation depending on a userutterance, according to an embodiment.

Referring to FIG. 14, the display device 100 may perform an operation ofproviding game result information and an operation of making viewingreservations.

According to an embodiment, in screen 1410, the display device 100 mayreceive a first user input for receiving game information called“dodgers baseball game result”. According to an embodiment, the displaydevice 100 may display a first UI 1403 (e.g., the first UI 310 a in FIG.3) for interacting with a user, on a screen on which an image 1401 ofcontent (e.g., the image 340 a in FIG. 3) is displayed. According to anembodiment, the display device 100 may display an indicator 1403 aindicating the state of receiving the first user input and a text 1403 bcorresponding to the first user input (e.g., a user utterance), on thefirst UI 1403.

According to an embodiment, in screen 1420, the display device 100 maydisplay a third UI 1405 (e.g., the third UI 330 a in FIG. 3) fordisplaying the result of the first user input, on a screen on which theimage 1401 of content is displayed. According to an embodiment, thedisplay device 100 may output a result 1405 a for providing gameinformation on the third UI 1405 based on the first user input.According to an embodiment, a text 1403 c for the reply saying that“Dodgers won, 6 to 1” to the first user input may be displayed on thefirst UI 1403, and the reply may be output through a speaker. Accordingto an embodiment, the display device 100 may display the indicator 1403a for indicating a state of outputting the reply, on the first UI 1403.

According to an embodiment, in screen 1430, the display device 100 mayreceive a second user input for making a viewing reservation saying that“Please making a viewing reservation for the next game”. For example,the second user input may include information insufficient to perform anoperation of viewing reservation. According to an embodiment, thedisplay device 100 may display a text 1403 d corresponding to the seconduser input (e.g., a user utterance), on the first UI 1403.

According to an embodiment, in screen 1440, the display device 100 maydisplay a second UI 1407 (e.g., the second UI 320 a in FIG. 3) fordisplaying a feedback on the first user input, on a screen on which theimage 1401 of content is displayed. According to an embodiment, thedisplay device 100 may display a feedback 1407 a including gameinformation based on the first user input, on the second UI 1407.According to an embodiment, the display device 100 may display the replysaying that “Which channel do you want to reserve?” to the second userinput, on the first UI 1403 and then may output the reply through aspeaker. According to an embodiment, the display device 100 may receivea third user input associated with game selection.

According to an embodiment, in screen 1450, when completing thereservation for viewing a game corresponding to the second user input,the display device 100 may display a message saying that “Reservation iscompleted” on the second UI 1205 and may output the message through aspeaker.

In a process of performing the specified function depending on a userutterance, the display device 100 of the invention described withreference to FIGS. 1 to 14 may display a UI for interacting with a useron a display and may display a UI for displaying the result ofperforming the specified operation to overlap with a part of the imageof content, thereby providing a user experience (UE) that performs thespecified function in a dialog form while minimizing the obscuration ofthe image of the content.

According to various embodiments, at least a part of an apparatus (e.g.,modules or functions thereof) or a method (e.g., operations) may be, forexample, implemented by instructions stored in a computer-readablestorage media in the form of a program module. The instruction, whenexecuted by a processor, may cause the processor to perform a functioncorresponding to the instruction. The computer-readable recording mediummay include a hard disk, a floppy disk, a magnetic media (e.g., amagnetic tape), an optical medium (e.g., a compact disc read only memory(CD-ROM) and a digital versatile disc (DVD), a magneto-optical media(e.g., a floptical disk)), an embedded memory, or the like. The one ormore instructions may contain a code made by a compiler or a codeexecutable by an interpreter.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

1. A display device comprising: a microphone; a display configured todisplay a screen including a plurality of layers; a memory configured tostore a plurality of application programs; and at least one processorconfigured to display a first user interface (UI) for interacting with auser on a first layer among the plurality of layers, to display a secondUI for displaying information obtained by performing the interaction ona second layer among the plurality of layers, and to display an image atleast partly overlapping with the first UI and the second UI on a thirdlayer among the plurality of layers, wherein the at least one processoris configured to: display the first UI on the display, in a process ofperforming the interaction through the microphone and in a process ofdisplaying a result of performing the interaction on at least one of thesecond UI and the third layer.
 2. The display device of claim 1, whereinthe processor is configured to: receive a first user input through themicrophone; and display a result corresponding to the first user inputon at least one of the second UI and the third layer.
 3. The displaydevice of claim 2, wherein the processor is configured to: display afourth layer on a third UI for displaying a feedback on the first userinput; receive a second user input associated with the feedback throughthe microphone; and display a result corresponding to the second userinput on at least one of the second UI and the third layer.
 4. Thedisplay device of claim 3, wherein the feedback includes at least oneobject capable of being selected depending on a user input, and whereinthe second user input is an input to select one of the at least oneobject.
 5. The display device of claim 3, wherein the processor isconfigured to: selectively display one of the second UI and the third UIon the display.
 6. The display device of claim 3, further comprising: aspeaker, wherein the processor is configured to: output the feedbackthrough the speaker.
 7. The display device of claim 1, wherein the firstUI includes an indicator for displaying a state of the display device,and wherein the processor is configured to: when receiving a user input,display the indicator in a first state; and when displaying a resultcorresponding to the user input on the second UI and the third layer,display the indicator in a second state.
 8. The display device of claim1, wherein the processor is configured to: execute at least oneapplication program among the plurality of application programs based onthe result of performing the interaction; display information calculatedusing the executed application program on the second UI; or display afourth UI of the executed application program on the third layer.
 9. Thedisplay device of claim 1, wherein the processor is configured to:perform the interaction through an external server processing a userutterance.
 10. A control method of a display device, the methodcomprising: displaying a first UI for interacting with a user on a firstlayer among a plurality of layers included in a screen displayed on adisplay; receiving a first user input through a microphone; anddisplaying a result corresponding to the first user input, on a secondUI for displaying information obtained by performing the interaction ona second layer among the plurality of layers or on a third layer onwhich an image at least partly overlapping with the first UI and thesecond UI is displayed.
 11. The method of claim 10, further comprising:displaying a third UI for displaying a feedback on the first user inputon a fourth layer among the plurality of layers to obtain the resultcorresponding to the first user input; receiving a second user inputassociated with the feedback through the microphone; and displaying aresult corresponding to the second user input on at least one of thesecond UI and the third layer.
 12. The method of claim 11, wherein thefeedback includes at least one object capable of being selecteddepending on a user input, and wherein the second user input is an inputto select one of the at least one object.
 13. The method of claim 11,further comprising: outputting the feedback through a speaker.
 14. Themethod of claim 10, further comprising: when receiving a user input,displaying an indicator in a first state; and when displaying a resultcorresponding to the user input on the second UI and the third layer,displaying the indicator in a second state.
 15. The method of claim 11,wherein the displaying of the result on at least one of the second UIand the third layer includes: executing at least one application programamong a plurality of application programs stored in a memory based on aresult of performing the interaction; and displaying informationcalculated using the executed application program on the second UI ordisplay a fourth UI of the executed application program on the thirdlayer.